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Abstract 


The  focus  of  this  report  is  the  implementation  of  Total  Quality  Management 
in  ten  DoD  organizations.  The  participating  organizations  were  all  identified  by  the 
Federal  Quality  Institute  as  either  winners  or  finalists  for  the  Productivity/Quality 
Improvement  Prototype  (QIP)  award  sponsored  by  DoD  and  the  Office  of 
Management  and  Budget  Qualitative  data  collected  included  interviews  with  either 
top  executives  or  TQM  coordinators,  documentation  of  quality  management  activities. 
A  questionnaire  survey  was  also  administered  to  the  executive  steering  committee  of 
each  organization  providing  a  self-assessment  of  eight  dimensions  of  quality 
management  practices.  The  report  describes  the  lessons  learned,  promising  practices 
and  the  results  of  the  self-assessment  survey  for  the  participating  organizations. 
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RESULTS  OF  SELF-RATINGS  ON  8  FACTORS  OF  QUALITY  NANAGENENT 
AT  THE  NAVAL  AVIONICS  CENTER 


OVERVIEW 

This  report  is  derived  from  Master's  thesis  research  conducted  by  Lt. 
Carolyn  Applegate  at  the  Naval  Postgraduate  School  (Applegate,  1991).  The  focus 
of  that  thesis  was  to  document  the  practices  and  experiences  of  ten  exemplary  DoD 
organizations  in  the  implementation  of  Total  Quality  Management.  Three  types  of 
data  were  collected  from  each  organization.  An  interview  was  conducted  with 
either  the  top  executive  (military  or  civilian)  or  TQM  coordinator;  documents 
describing  TQM  planning,  achievements,  activities  were  collected;  and  a 
Questionnaire  survey  was  distributed  to  the  executive  steering  committee  at  each 
location  to  evaluate  self-assessments  of  specific  quality  management  practices. 
The  thesis  and  a  subsequent  technical  report  (Applegate,  Hocevar  &  Thomas,  1991) 
describe  the  results  of  both  qualitative  and  quantitative  data  analyses 
presenting  both  lessons  learned  and  promising  practices  for  the  entire  sample  of 
organizations. 

The  focus  of  this  companion  report  is  on  the  results  of  the  quantitative 
survey  data  gathered  from  one  of  the  participating  organizations  --  the  Naval 
Avionics  Center  (NAC).  As  described  in  the  larger  study,  self-assessment  of 
management  practices  can  be  a  valuable  tool  in  the  achievement  of  quality 
improvements.  The  intent  of  this  report  is  to  provide  NAC  with  data  representing 
the  top  management  group's  self-assessment  of  the  organization's  quality 
management  practices  for  their  use  in  planning  their  continuous  improvement 
strategies. 
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BACKGROUND 

For  the  past  few  years,  top  executives  in  a  number  of  industries  have  been 
rethinking  how  to  measure  quality  performance.  During  the  1980s,  many  managers 
involved  in  the  quality  movement  came  to  realize  that  quality  is  a  strategic 
weapon  in  a  competitive  world;  this  resulted  in  new  performance  measures  such  as 
tracking  defect  rates  and  response  times  (Troxell,  1981).  The  impetus  of  growth 
of  the  Total  Quality  concept,  development  of  the  Malcolm  Baldridge  National 
Quality  Award,  and  increasingly  stringent  manufacturer  demands  on  quality  of 
supplier  goods  have  led  to  a  broadening  of  performance  measures  through  an 
emphasis  on  quality.  (Eccles,  1991) 

One  problem  with  these  new  performance  measurements  is  that  relying  on 
measurements  of  customer  satisfaction,  quality,  and  innovation  is  not  as  well 
ingrained  in  today's  managers  as  financial  performance  measures.  Current 
information  resources  do  not  readily  support  real-time  management  using  new 
quality  measures  because  they  were  designed  based  on  traditional  accounting 
systems.  Real-time,  operational  measures  of  quality  management,  which  broaden 
the  basis  of  organizational  performance  measurement,  can  aid  decision-makers  to 
influence  critical  areas  such  as  process  management  in  order  to  improve 
performance.  {Goldra*t  and  Cox,  1986) 

Most  organizations  which  use  statistical  process  control  tools  collect 
performance  data  such  as  rework  or  defect  rates  that  focus  on  production. 
However,  these  measures  are  limited  in  that  they  do  not  reflect  organization-wide 
quality  management.  Various  authors  on  quality  recommend  principles  for 
effectively  managing  quality.  These  include  Deming  (1982;1986),  Juran's  (1986) 
quality  trilogy,  Crosby's  (1979)  zero-defect  improvement  programs,  Ishikawa's 
(1985)  total  quality  control,  and  Leonard  and  Sasser's  (1982)  identification  of 
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quality  levers.  It  is  notable  that  all  of  these  authors  discuss  the  ideals  of 
top  management  commitment,  education,  continuous  improvement,  and  employee 
involvement.  Examination  of  these  and  other  principles  provides  a  foundation  for 
recognizing  areas  critical  to  any  change  in  quality  focus. 

The  first  thorough  and  systematic  attempt  to  synthesize  some  of  these 
quality  concepts  is  shown  in  Table  1,  adapted  from  a  previous  study  (Saraph, 
Benson  and  Schroeder,  1989).  Building  on  the  writings  of  quality  management 
authors,  Saraph  et  al .  propose  organizational  requirements  for  effective  quality 
management.  These  organizational  requirements  are  classified  into  eight  critical 
factors  necessary  to  achieve  a  successful  shift  to  a  quality  focus.  They 
include:  the  role  of  management,  leadership  and  quality  policy;  the  role  of  the 
quality  department;  training;  product  and  service  design;  supplier  quality 
management;  process  management;  quality  data  and  reporting;  and  employee 
relations.  Measures  of  these  critical  factors  can  form  a  profile  of  an 
organization's  quality  management  practices  that  provides  a  benchmark  for  making 
decisions  to  achieve  higher  or  more  ideal  levels  of  quality  within  an 
organization.  The  eight  critical  factors  and  an  explanation  of  what  they 
represent  are  shown  in  Table  2. 

METHODOLOGY 

Choice  of  Organizations 

The  DOD/Office  of  Management  and  Budget's  (OMB)  Productivity/Quality 
Improvement  Prototype  (QIP)  was  selected  as  the  criterion  for  research 
participation.  The  purpose  of  this  award  Is  to  recognize  early  successes, 
provide  models  for  productivity  improvement  in  other  agencies,  and  provide 
visibility  for  high  achievers.  The  Federal  Quality  Institute  was  contacted  in 
order  to  develop  a  list  of  QIP  winners  and  finalists  since  the  award's  inception 
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Table  2:  CRITICAL  FACTORS  OF  QUALITY  MANAGEMENT 
(adapted  from  Saraph  et  al.,1989) 


Critical  Factors 

of  Quality  Management _ Explanation  of  Critical  Factors 


1.  Role  of 
management 
leadership  and 
quality  policy 

Acceptance  of  quality  responsibility  by  top 
management  and  department  heads.  Evaluation  of 
top  management  on  quality.  Participation  by  top 
management  in  quality  improvement  efforts. 
Specificity  of  quality  goals.  Importance 
attached  to  quality  in  relation  to  cost  and 
schedule.  Comprehensive  quality  planning. 

2.  Role  of  the 
quality 
department 

Visibility  and  autonomy  of  the  quality 
department.  Quality  department's  access  to  top 
management.  Use  of  quality  staff  for 
consultation.  Coordination  between  quality 
department  and  other  departments.  Effectiveness 
of  the  quality  department. 

3.  Training 

Provision  of  statistical  training,  trade 
training,  and  quality-related  training  for  all 
employees. 

4.  Product /service 
design 

Thorough  scrub-down  process.  Involvement  of  all 
affected  departments  in  design  reviews. 

Emphasis  on  producibility.  Clarity  of 
specifications.  Emphasis  on  quality,  not  roll¬ 
out  schedule.  Avoidance  of  frequent  redesigns. 

5.  Supplier  quality 
management 

Fewer  dependable  suppliers.  Reliance  on 
supplier  process  control.  Strong 
interdependence  of  supplier  and  customer. 
Purchasing  policy  emphasizing  quality  rather 
than  price.  Supplier  quality  control.  Supplier 
assistance  in  product  development. 

6.  Process 

management 

Clarity  of  process  ownership,  boundaries,  and 
steps.  Less  reliance  on  inspection.  Use  of 
statistical  process  control.  Selective 
automation.  Fool-proof  process  design. 

Preventive  maintenance.  Employee  self- 
inspection.  Automated  testing. 

7.  Quality  data  and 
reporting 

Use  of  quality  cost  data.  Feedback  of  quality 
data  to  employees  and  managers  for  problem 
solving.  Timely  quality  measurement.  Evaluation 
of  managers  and  employees  based  on  quality 
performance.  Availability  of  quality  data. 

8 .  Employee 
relations 

Implementation  of  employee  involvement  and 
quality  circles.  Open  employee  participation  in 
quality  decisions.  Responsibility  of  employees 
for  quality.  Employee  recognition  for  superior 
quality  performance.  Effectiveness  of 
supervision  in  handling  quality  issues.  On¬ 
going  quality  awareness  of  all  employees. 
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in  1988.  The  resulting  list  identified  23  organizations,  11  of  which  were 
within  DOD.  All  the  DOO  organizations  were  contacted  and  10  agreed  to 
participate.  Each  DOD  organization  provided  a  point  of  contact 
responsible  for  all  administration  concerned  with  this  study.  A  list  of 
the  participating  organizations  is  given  in  Appendix  A. 

Questionnaire  Survey 

A  survey  was  adapted  from  a  private  sector  study  which 
developed  and  validated  an  instrument  to  measure  the  critical  factors  of 
quality  management  (Saraph,  Benson  and  Schroeder,  1989).  As  described  in 
the  background  section,  the  original  survey  developers  based  the  questions 
on  the  current  literature  on  quality  management  including  such  authors  as 
Doming,  Juran,  Ishikawa,  and  Crosby.  The  questionnaire  was  validated 
using  a  sample  from  private  sector  organizations  in  Minneapolis,  Minnesota 
including  3M,  Control  Data,  and  Northwest  Airlines. 

The  adapted  form  of  the  survey  contains  66  questions  composing  the 
eight  critical  factors  that  assess  a  manager's  perception  of  actual 
quality  practices  within  his/her  organization.  Question  wording  was 
minimally  modified  to  fit  DoD  organizations  (e.g.,  "top  executive"  was 
changed  to  "commanding  officer  or  executive  director").  The  modified 
survey  as  it  was  administered  for  this  study  Is  shown  In  Appendix  B. 
Additional  Information  on  the  reliability  and  validity  of  the  survey 
Instrument  can  be  found  In  Appendix  C. 

A  typical  survey  Item,  as  shown  below,  allows  managers  to  Indicate 
their  perception  of  the  degree  or  extent  of  a  given  practice  within  their 
organization: 
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Extent  or  Degree  of  Current  Practice  Is 
Very  low  Low  Medium  High  Very  High 
amount  of  final  inspection,  1234  5 

review  or  checking 

Survey  respondents  were  instructed  to  circle  the  number  that  represented 
their  perception  of  quality  management  practices  in  their  organization. 
Each  critical  factor  was  assessed  using  several  component  questions.  For 
each  component  question  and  for  each  critical  factor,  the  actual  level  of 
practice  within  or  across  organizations  is  represented  by  the  average  of 
the  respondents'  ratings  for  the  component  question  or  critical  factor. 
The  scale  scores  were  calculated  by  summing  the  component  item  ratings  and 
dividing  by  the  number  of  items.  The  items  comprising  each  critical 
factor  along  with  the  coefficient  alpha  statistic  of  internal  consistency 
reliability  are  presented  in  Table  3. 

Survey  Administration 

The  survey  respondents  chosen  within  the  ten  organizations 
were  members  of  each  organization's  quality  council  or  executive  steering 
committee,  because  these  people  serve  to  lead  the  quality  focus  within 
each  organization.  Survey  data  was  collected  in  May,  1991.  Each  survey 
respondent  assessed  the  degree  or  extent  of  actual  quality  management 
practices  in  his/her  organization  according  to  the  measure  described 
above.  Table  4  lists  the  ten  organizations  anonymously,  along  with  the 
number  of  responses  anticipated  and  the  number  of  survey  responses 
received.  At  MAC  (organization  #2  In  Table  4),  eleven  members  of  the 
Continuous  Improvement  Council  participated  in  the  study. 

Data  Analysis 

Data  were  analyzed  both  In  terms  of  Individual  items  and  the 
composite  scales  representing  each  of  the  eight  critical  factors  of 
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Table  3:  INTERNAL  CONSISTENCY  RELIABILITY  ANALYSIS  RESULTS 
_ FOR  THE  CRITICAL  FACTORS  OF  QUALITY  MANAGEMENT _ 

Critical  Factors  and  component  questions  _  CoefTicient  Alpha 

Roto  of  top  manogomonl  toattorthlp  and  quality  policy 
•oxtont  to  whict)  ttie  top  executivo  assumes  responsibility  for  quality  performance 
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Table  4:  SURVEY  RESPONSE 

Organization 

Surveys 

received 

Surveys 

sent 

Response 

rate 

#1 

7 

7 

100% 

#2 

11 

13 

85% 

#3 

14 

25 

56% 

#4 

10 

15 

67% 

#5 

10 

12 

83% 

#6 

11 

15 

73% 

#7 

20 

27 

74% 

#8 

8 

8 

100% 

#9 

6 

12 

50% 

#10 

5 

8 

63% 

Totai 

102 

142 

72% 

quality  management.  Scale  scores  were  calculated  by  summing  the  items 
comprising  each  scale  and  dividing  by  the  number  of  items.  This  allows 
the  scale  scores  to  retain  the  same  units  of  measure  for  ease  of 
comparability. 

Two  sets  of  analyses  were  performed  on  these  data.  The  first 
analysis  addressed  NAC  data  solely  and  provides  for  all  questions  in  the 
survey  both  the  mean  rating  and  the  frequency  of  responses  in  each  rating 
category.  The  mean  rating  represents  the  overall  attitude  of  the  group  on 
the  degree  of  practice  for  each  particular  aspect  of  quality  management. 
The  frequency  demonstrates  the  degree  of  agreement  that  exists  within  the 
group  on  the  status  of  managerial  practice  for  each  item. 

The  second  analysis  compared  the  self-ratings  of  NAC  with  those  from 
the  remaining  DoD  organizations  for  all  of  the  individual  items  comprising 
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the  eight  critical  factors  of  quality  management.  This  comparison  uses  a 
t-test  comparison  of  means  and  a  critical  t-value  appropriate  for  a 
significance  level  of  .10.  When  the  variances  of  the  two  groups  (NAC  and 
other  DoD)  were  not  statistically  different,  the  pooled  variance  estimate 
for  the  t-value  and  significance  level  was  used.  If  the  variances  were 
statistically  different,  the  separate  variance  estimate  was  used. 

RESULTS  EXAMINING  NAC  DATA  ONLY 

Tables  5-12  present  the  mean  or  average  rating  of  NAC  participants 
for  each  item  in  the  eight  critical  factors  of  quality  management.  Also 
shown  are  the  frequency  of  responses  for  each  rating  category.  The  items 
are  organized  according  to  the  8  factors.  The  presentation  of  items  is 
from  highest  to  lowest  rated  within  each  factor  category.  In  interpreting 
these  means,  note  that  a  value  of  "3"  represents  a  moderate  rating  of  the 
extent  of  current  practice.  It  is  important  to  note  that  the  ratings 
reflect  the  executive  steering  committee's  self-evaluation  of  current  TQM 
practices.  In  other  words,  the  data  reflect  how  the  top  management  at  NAC 
think  they  are  doing  on  specific  TQM-related  activities.  There  are  no 
norms  for  evaluating  "good"  or  "poor"  performance.  The  value  of  these 
data  is  in  guiding  discussion  and  critical  analysis  of  the  implications 
for  action  planning. 

Factor  #1:  Role  of  Top  Management  Leadership  and  Quality  Policy 
The  means  and  frequency  of  responses  for  factor  #1  are 
presented  in  Table  5.  Within  this  factor,  "Role  of  top  management 
leadership  and  quality  policy,"  the  most  highly  rated  items  are  "the 
extent  to  which  top  management  supports  long-term  quality  improvement 
process,"  "the  degree  of  participating  by  major  branch  or  department  heads 
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Table  5 

Factor  #1:  Role  of  Top  Nanagement  Leadership  and  Quality  Policy 
Frequencies  and  Neans  of  NAC  Self-Ratings 


Rating  Mean 

5  4  3  2  1  _ 

56000  4.5 

27200  4.0 

53201  4.0 

35120  3.8 

35111  3.7 

14510  3.5 

07211  3.4 

23321  3.3 

11711  3.0 

05231  3.0 

04331  2.9 

01730  2.8 

01730  2.8 

3.4 


Item 


extent  to  which  top  management  supports  long-term 
quality  improvement  process 

degree  of  participation  by  major  branch  or 
department  heads  in  the  quality  improvement  process 

degree  to  which  top  management  considers  quality 
management  as  a  way  to  increase  revenues/reduce 
costs 

extent  to  which  the  top  executive  assumes 
responsibility  for  quality  performance 

importance  attached  to  quality  by  top  management  in 
relation  to  cost  and  schedule  objectives 

acceptance  of  responsibility  for  quality  by  major 
branch/department  heads  within  the  organization 

extent  to  which  top  management  has  objectives  for 
quality  performance 

amount  of  review  of  quality  issues  in  top  management 
meetings 

degree  of  comprehensiveness  of  the  quality  plan 
within  the  organization 

degree  to  which  top  management  (commanding 
officer/executive  director/major  department  heads) 
is  evaluated  for  quality  performance 

specificity  of  quality  goals  within  the  organization 

extent  to  which  quality  goals  and  policy  are 
understood  within  the  organization 

comprehensiveness  of  the  goal -setting  process  for 
quality  within  the  organization 

Overall  Score  for  Critical  Factor  #  1 
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In  the  quality  improvement  process,"  and  "the  degree  to  which  top 
management  considers  quality  management  as  a  way  to  increase 
revenues/reduce  costs."  Each  of  these  three  items  has  a  mean  rating  of 
greater  than  4.0.  The  aspects  of  factor  #1  that  receive  the  lowest  self- 
rating  all  relate  to  quality  goals.  The  specificity  of  quality  goals,  the 
extent  to  which  these  goals  are  understood  within  the  organization,  and 
the  comprehensiveness  of  the  goal -setting  process  for  quality  all  receive 
ratings  below  3.0. 

Factor  #2:  Role  of  the  Quality  Department 

All  five  items  comprising  this  section  (see  Table  6)  have  mean 
ratings  of  at  least  3.5,  indicating  a  positive  view  of  the  managerial 
practices  related  to  the  quality  department.  The  highestt  rating  is  for 
the  quality  department's  access  to  top  management.  The  Imest  ratings  in 
this  section  are  for  the  autonomy  of  the  department  /[x»3.6)  and  the 
effectiveness  of  the  department  in  improving  quality  (/=3.5). 

Factor  #3:  Training  / 

Two  items  in  the  training  factor  are  rated  the  highest  in  this 
category  (see  Table  7)  with  means  greater  than  4.0.  These  are  "commitment 
of  top  management  to  employee  training"  and  "training  in  the  total  quality 
concept. . .throughout  the  organization."  The  two  items  with  the  lowest 
self-ratings  pertain  to  quality- related  training  given  to  non-supervisory 
employees,  and  training  in  advanced  statistical  techniques  with  means  of 
3.2  and  2.7,  respectively. 

Factor  #4:  Product/Service  Design 

The  range  of  means  within  this  factor  of  quality  management  is 
fairly  narrow  (see  Table  8).  The  highest  self-rating  is  for  coordination 
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Table  6 

Factor  #2:  Role  of  Quality  Department 
Frequencies  and  Keans  of  NAC  Self-Ratings 


Rating 

5  4  3  2  1 

Mean 

Item 

8  3  0  0  0 

4.7 

quality  department's  access  to  top  management 

3  5  3  0  0 

4.0 

visibility  of  the  quality  department 

2  5  4  0  0 

3.8 

amount  of  coordination  between  the  quality 

department  and  other  departments 

2  6  111 

3.6 

autonomy  of  the  quality  department 

0  6  4  1  0 

3.5 

effectiveness  of  the  quality  department  in  improving 

qual ity 

3.9 

Overall  Score  for  Critical  Factor  #  2 
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Table  7 

Factor  #3:  Training 

Frequencies  and  Neans  of  MAC  Self-Ratings 


Rating 
5  4  3  2  1 

4  7  0  0  0 

4  6  10  0 

3  4  3  1  0 
3  3  4  1  0 

2  4  5  0  0 

1  4  6  0  0 

0  3  7  1  0 

0  3  3  4  1 


Mean 


Item 


4.4  commitment  of  the  top  management  to  employee 

training 

4.3  training  in  the  "total  quality  concept"  (i.e. 

philosophy  of  organization-wide  responsibility  for 
quality)  throughout  the  organization 

3.8  availability  of  resources  for  employee  training 

3.7  quality-related  training  given  to  managers  and 

supervisors  throughout  the  organization 

3.7  training  in  the  basic  statistical  techniques  (such 
as  histograms  and  control  charts)  in  the 
organization  as  a  whole 

3.5  specific  work-skills  training  (technical  and 
vocational)  given  to  non-supervisory  employees 
throughout  the  organization 

3.2  quality-related  training  given  to  non-supervisory 
employees  throughout  the  organization 

2.7  training  in  advanced  statistical  techniques  (such  as 
design  of  experiments  and  regression  analysis)  in 
the  organization  as  a  whole 

3.7  Overall  Score  for  Critical  Factor  #  3 
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Table  8 

Factor  #4:  Product/Service  Design 
Frequencies  and  Means  of  MAC  Self-Ratings 


Rating 
5  4  3  2  1 

14  5  10 
0  6  4  1  0 
1  3  5  2  0 
0  1  10  0  0 
0  2  7  2  0 

0  18  2  0 


Mean 


Item 


3.5  coordination  among  affected  departments  in  the 
process/service  development  process 

3.5  quality  of  new  processes/services  emphasized  in 

relation  to  cost  or  schedule  objectives 

3.3  extent  to  which  implementation/producibility  is 

considered  in  the  process/service  design  process 

3.1  clarity  of  process/service  specifications  and 
procedures 

3.0  thoroughness  of  new  process/service  design  reviews 
before  the  process/service  is  implemented/produced 

2.9  quality  emphasis  by  customer  service  employees 

3.2  Overall  Score  for  Critical  Factor  #  4 
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among  departments  in  the  development  process  (x=3.5);  while  the  lowest  is 
for  the  quality  emphasis  by  customer  service  employees  (x=2.9).  As  with 
the  interpretation  of  all  results,  the  implications  of  these  ratings 
depends  on  the  goals  and  capabilities  of  the  organization.  Within  this 
factor,  all  ratings  are  clustered  fairly  closely  to  the  midpoint  of  the 
rating  scale,  indicating  that  the  executive  group  perceives  a  moderate 
degree  of  practice  of  these  specific  management  activities  in  support  of 
quality.  Whether  this  represents  a  satisfactory  achievement  of  quality 
practices  in  this  area  or  becomes  a  benchmark  for  increased  attention  to 
these  activities  depends  on  the  organizations's  strategies  for  improving 
quality. 

Factor  #5:  Supplier  Quality  Hanagement 

Of  the  eight  items  comprising  this  critical  factor,  only  three 
show  means  greater  than  or  equal  to  the  midpoint  of  3.0  (see  Table  9). 
These  include:  "extent  to  which  suppliers  are  selected  based  on  quality 
rather  than  price  or  schedule"  {x=3.5),  "thoroughness  of  the  supplier 
rating  system"  (x=3.2),  and  "clarity  of  specifications  provided  to 
suppliers"  (x=3.0).  The  item  with  the  lowest  mean  rating  of  1.8  pertains 
to  the  involvement  of  the  supplier  in  the  product  development  process. 

Factor  #6:  Process  Hanagemert 

The  items  that  receive  the  highest  rating  in  this  factor 
category  (see  Table  10)  all  relate  to  aspects  of  inspection  (incoming, 
acceptance  sampling,  final,  and  in-process).  The  means  for  these  four 
items  range  from  3.4  to  3.7  and  indicate  that  the  practice  of  Inspections 
is  perceived  to  be  above  the  "moderate"  level  by  the  executive  group  at 
NAC.  The  issue  of  the  role  of  inspections  within  the  philosophy  of  TQM, 
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Rating 
5  4  3  2  1 

15  4  10 

2  15  3  0 
0  2  7  2  0 
0  2  6  3  0 
0  2  3  4  1 

0  2  4  3  2 
0  2  3  5  1 

0  0  2  4  4 


Table  9 

Factor  #5:  Supplier  Quality  Nanagement 
Frequencies  and  Means  of  MAC  Self-Ratings 

Mean  Item 


3.5  extent  to  which  suppliers  are  selected  based  on 

quality  rather  than  price  or  schedule 

3.2  thoroughness  of  the  supplier  rating  system 

3.0  clarity  of  specifications  provided  to  suppliers 

2.9  technical  assistance  provided  to  suppliers 

2.6  amount  of  education  of  suppliers  by  the 

organization 

2.5  reliance  on  reasonably  few  dependable  suppliers 

2.5  extent  to  which  longer  term  relationships  are 

offered  to  suppliers 

1.8  involvement  of  the  supplier  in  the  product 

development  process 

2.7  Overall  Score  for  Critical  Factor  #  5 
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Table  10 

Factor  #6:  Process  Management 
Frequencies  and  Means  of  MAC  Self-Ratings 


Rating  Mean  Item 

5  4  3  2  1  _  _ 

1  5  4  0  0  3.7  use  of  acceptance  sampling  to  accept/reject  lots  or 

batches  of  work 

25301  3.6  amount  of  Incoming  Inspection,  review  or  checking 

16301  3.5  amount  of  final  inspection,  review  or  checking 

05510  3.4  amount  of  in-process  Inspection,  review  or 

checking 

04520  3.2  amount  of  preventive  equipment  maintenance 

02711  2.9  degree  of  automation  of  the  process 

02540  2.8  stability  of  production  schedule/work  distribution 

0  0  8  3  0  2.7  clarity  of  work  or  process  Instructions  given  to 

employees 

01721  2.7  extent  to  which  Inspection,  review  or  checking  of 

work  is  automated 

0  0  4  7  0  2.4  extent  to  which  the  design  is  "fool-proof"  and 

minimizes  chances  of  employee  errors 

3.1  Overall  Score  for  Critical  Factor  #  6 
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particularly  as  applied  by  DoD  organizations  is  hotly  debated.  Therefore, 
it  is  difficult  to  interpret  these  numbers.  If  NAC  has  determined  a 
certain  level  of  inspection  is  critical  to  operations,  the  ratings  on 
these  items  may  reflect  the  achievement  of  organizational  objectives  in 
this  area.  If,  however,  the  organization  is  attempting  to  decrease  its 
reliance  on  inspections,  the  ratings  may  reflect  an  area  needing  increased 
attention. 

Three  of  the  items  in  this  factor  category  that  have  ratings  below 
3.0  relate  to  process  activities  that  can  diminish  the  reliance  on 
inspections  for  quality.  These  include  the  stability  of  production 
schedule  and  work  distribution  (x=2.8),  the  clarity  of  work  or  process 
instructions  given  to  employees  (x*2.7),  and  the  extent  to  which  the 
design  is  "fool-proof"  and  minimizes  chances  of  employee  errors  {x=2.4). 

Factor  #7:  Quality  Data  and  Reporting 

The  ratings  within  this  factor  category  (see  Table  11)  support 
the  findings  of  the  larger  study  from  which  these  data  are  drawn 
(Applegate,  1991;  Applegate,  Hocevar  &  Thomas,  1991).  That  study  reports 
the  area  of  quality  data  is  one  of  the  most  challenging  being  faced  by  all 
of  the  exemplary  DoD  organizations  who  participated.  The  self-rating  data 
for  NAC  show  that  only  two  of  the  eight  items  have  means  above  the 
midpoint  --  availability  of  quality  data  and  timeliness  of  quality  data 
(3.2  and  3.0,  respectively).  The  remaining  items  have  means  ranging  from 
2.5  to  2.9.  These  ratings  demonstrate  that  the  extent  to  which  quality 
data  is  used  for  quality  management,  is  available  to  or  displayed  for  non- 
supervisory  employees,  or  is  used  to  evaluate  managerial  performance  are 
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Table  11 

Factor  #7:  Quality  Data  and  Reporting 
Frequencies  and  Heans  of  NAC  Self -Ratings 


Rating 
5  4  3  2  1 

1  2  6  2  0 

0  3  5  3  0 
0  2  7  1  1 

0  3  3  5  0 

0  15  5  0 

0  2  5  2  2 

0  2  4  3  2 
0  14  5  1 


Mean  Item 


3.2  availability  of  quality  data  (error  rates,  defect 
rates,  scrap,  defects) 

3.0  timeliness  of  the  quality  data 

2.9  extent  to  which  quality  data  are  available  to 

managers  and  supervisors 

2.8  extent  to  which  quality  data  (cost  of  quality, 

defects,  errors,  scrap,  etc.)  are  used  as  tools  to 
manage  quality 

2.6  extent  to  which  quality  data  are  available  to  non- 
supervisory  employees 

2.6  extent  to  which  quality  data,  control  charts,  etc., 
are  displayed  at  employees'  workstations 

2.5  extent  to  which  quality  data  are  used  to  evaluate 
supervisor  and  managerial  performance 

2.5  availability  of  cost  of  quality  data  in  the 

organization 

2.8  Overall  Score  for  Critical  Factor  #  7 


22 


practices  that  NAC  may  want  to  discuss  as  possible  actions  for  improving 
the  effective  use  of  quality  data. 

Factor  #8:  Employee  Relations 

The  final  quality  management  category  shows  a  range  of  self- 
ratings  from  2.5  to  3.5  (see  Table  12).  The  extent  to  which  quality 
awareness  building  is  ongoing  among  employees  and  the  extent  to  which 
employee  involvement  type  programs  are  implemented  both  have  mean  ratings 
of  3.5.  The  lowest  mean  rating  (2.5)  is  for  the  extent  to  which  employees 
are  held  responsible  for  error- free  output.  The  amount  of  feedback 
provided  employees  on  their  quality  performance  and  the  degree  of 
participation  in  quality  decisions  by  non-supervisory  employees  both 
receive  mean  ratings  of  2.7.  The  value  of  these  results  is  that  they 
focus  on  specific  aspects  of  employee  involvement  in  the  quality 
improvement  process.  Thus,  these  practices  can  be  discussed  explicitly  by 
the  executive  group  as  to  their  potential  value  and  the  best  mechanisms 
for  implementing  them  if  such  a  decision  should  be  made, 

RESULTS  COMPARING  NAC  WITH  OTHER  OOD  EXEMPLARY  ORGANIZATIONS 

The  preparation  and  presentation  of  results  comparing  NAC's  self- 
evaluation  with  the  self-evaluation  of  the  other  exemplary  DoD 
organizations  was  done  at  the  request  of  NAC's  executive  steering  group. 
There  are  several  cautions  that  should  be  mentioned  before  describing 
these  results.  First,  these  data  do  not  represent  objective  performance 
ratings,  but  the  self-evaluation  of  organizational  members.  For  this 
reason,  the  interpretation  of  comparative  results  is  difficult.  It  is 
possible,  for  example,  that  within  the  DoO  sample  there  may  be  inflated 
self-perceptions  as  a  result  of  having  recently  won  the  quality  award  -- 
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Table  12 

Factor  #8:  Employee  Relations 
Frequencies  and  Means  of  MAC  Self-Ratings 


Rating 
5  4  3  2  1 

0  6  4  1  0 

14  5  10 

1  2  5  2  1 

0  2  8  0  1 

0  3  5  3  0 

0  17  2  1 
0  0  8  3  0 

0  0  7  2  2 


Mean 


Item 


3.5  extent  to  which  quality  awareness  building  among 
employees  Is  ongoing 

3.5  extent  to  which  quality  circle  or  employee 
involvement  type  programs  are  implemented  in  the 
organization 

3.0  extent  to  which  employees  are  recognized  for 
superior  quality  performance 

3.0  effectiveness  of  supervisors  in  solving 
problems/issues 

3.0  effectiveness  of  the  quality  circle  or  employee 
involvement  type  programs  in  the  organization 

2.7  amount  of  feedback  provided  to  employees  on  their 
quality  performance 

2.7  degree  of  participation  in  quality  decisions  by 
non-supervisory  employees 

2.5  extent  to  which  employees  are  held  responsible  for 
error-free  output 

3.0  Overall  Score  for  Critical  Factor  #  8 
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"We  just  'won' I  We  must  be  doing  great!"  This  phenomenon  is  referred  to 
as  retrospective  rationality  (Pfeffer,  1982)  and  can  lead  to  exaggerated 
differences  between  the  self-ratings  of  the  larger  group  that  includes 
some  award  winners,  and  the  self-ratings  of  NAC  that  may  be  casting  a 
critical  eye  on  its  own  performance  in  trying  to  enhance  quality 
operations.  However,  the  influence  of  retrospective  rationality  is  only 
speculative;  there  is  no  way  to  evaluate  the  "accuracy"  of  the  self-report 
data  within  the  design  constraints  of  this  study. 

A  second  caution  follows  the  warning  of  TQM  advocates  against 
seeking  a  "recipe  for  success"  by  following  the  model  of  another 
organization.  To  the  extent  that  the  achievement  of  other  organizations 
can  inspire  NAC  to  further  pursue  aspects  of  quality  management,  these 
comparative  results  may  prove  to  be  motivational.  There  are  potential 
benefits  to  the  competition  inherent  in  comparison  data;  but  competition 
can  also  be  dysfunctional  if  the  pursuit  of  a  certain  activity  is 
undertaken  because  others  are  rating  themselves  higher  in  that  area  rather 
than  because  it  is  determined  to  be  the  appropriate  activity  given  the 
organization's  own  analyses  of  quality  processes. 

In  addition  to  the  increased  motivation  mentioned  above,  another 
potential  value  of  the  comparative  data  is  in  highlighting  areas  where 
other  organizations  may  have  developed  innovative  practices.  Such 
innovative  practices  are  briefly  described  and  contact  points  are 
identified  in  the  full  reports  of  this  research  project  (see  Applegate, 
1991;  or  Applegate,  Hocevar  &  Thomas,  1991).  A  final  value  of  the 
comparative  results  is  in  identifying  areas  where  the  challenge  of 
implementing  quality  management  is  experienced  by  other  exemplary 


25 


organizations.  In  other  words,  practices  where  NAC's  self-rating  is  low 
may  also  be  reported  as  low  by  other  organizations,  indicating  the 
difficulty  inherent  in  the  changes  required  by  continuous  improvement. 
This  occurs  in  a  number  of  items  where  there  is  no  significant  difference 
between  NAC's  self-rating  and  the  self-ratings  of  the  other  DoD 
organizations  on  items  identified  as  problematic  in  the  review  of 
previously  described  results. 

Overview  of  Comparative  Findings 

The  t-test  comparison  of  means  for  all  66  items  was  conducted 
with  NAC  (N=ll)  and  the  remaining  DoD  organizational  participants  (N=91) 
as  the  two  comparison  groups.  The  two  groups  were  found  to  have  self- 
ratings  that  were  not  significantly  different  for  more  than  70%  of  the 
items.  This  indicates  that  in  most  of  the  areas  of  quality  management 
practices,  NAC's  self-rating  of  the  extent  of  current  practice  is  the  same 
as  the  self-rating  of  other  exemplary  DoD  organizations. 

Practices  Where  NAC  Ratings  Here  Higher  Than  DoD  Ratings 

The  practices  where  NAC's  self-ratings  were  significantly 
higher  than  the  DoD  comparison  group  are  presented  in  Table  13;  the  items 
are  all  from  two  categories  of  quality  management  --  supplier  quality 
management  and  process  management.  In  the  first  category,  the  extent  to 
which  suppliers  are  selected  on  the  basis  of  quality  rather  than  price  or 
schedule  and  the  thoroughness  of  the  supplier  rating  system  show  NAC  with 
mean  ratings  greater  than  the  DoD  comparison  group.  This  result  is  not 
surprising  given  the  Blue  Ribbon  Contractor  program  that  has  been 
developed  at  NAC  and  is  documented  in  the  larger  report  as  an  exemplary 
practice  (see  Applegate,  1991;  Applegate,  Hocevar  &  Thomas,  1991). 
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Table  13 


Quality  Management  Items  Showing 
MAC  Self-Ratings  as  Higher^  than 
Other  OoD  Benchmark  Organizations 


Mean  Ratings 

Item 

NAC 

DoD 

(N=ll)  (N*91) 

5. 

Supplier  quality  management 

3.5 

2.2 

extent  to  which  suppliers  are  selected  based  on  quality 
rather  than  price  or  schedule 

3.2 

2.4 

thoroughness  of  the  supplier  rating  system 

6. 

Process  management 

3.7 

2.7 

use  of  acceptance  sampling  to  accept/reject  lots  or 
batches  of  work 

3.6 

2.8 

amount  of  incoming  inspection,  review  or  checking 

3.5 

3.0 

amount  of  final  Inspection,  review  or  checking 

2.7 

2.3 

extent  to  which  inspection,  review  or  checking  of  work  is 
automated 

^  Differences  were  determined  using  a  t-test  comparison  of 
mean  ratings.  All  those  reported  are  significant  p<.lO.  All  items 
not  reported  showed  no  difference  between  NAC  and  the  other 
participating  organizations. 
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The  second  category  of  process  management  has  four  items  where 
NAC's  mean  ratings  were  significantly  greater  than  the  comparison  group. 
These  items  a11  relate  to  inspections.  As  discussed  in  the  description  of 
results  that  examined  solely  NAC  data,  the  interpretation  of  these 
findings  is  problematic.  If  the  goal  of  the  organization  is  to  decrease 
reliance  on  inspections,  this  difference  may  indicate  that  NAC,  while 
scoring  higher,  is  not  as  far  along  as  the  other  organizations  perceive 
themselves  to  be  in  achieving  this  quality  aim.  However,  it  is  also 
possible  that  NAC  has  determined  that  inspections  cannot  be  decreased  due 
to  the  nature  of  the  process  or  product.  The  interpretation  and 
determination  of  subsequent  actions  based  on  these  results  is 
fundamentally  the  responsibility  of  organizational  members. 

Practices  Where  NAC  Ratings  Were  Lower  Than  DoD  Ratings 

The  Items  showing  NAC  with  mean  self-ratings  lower  than  the 
DoD  comparison  group  are  presented  in  Table  14.  Several  of  the  items 
included  in  these  results  have  already  been  identified  as  potentially 
problematic  in  the  discussion  of  results  for  the  NAC  data  alone.  For 
example,  within  the  first  category,  the  two  items  on  the  comprehensiveness 
of  the  goal  setting  process  for  quality  and  the  extent  of  understanding 
within  the  organization  of  quality  goals  and  policies  were  found  to  have 
the  lowest  mean  self-ratings  based  on  the  examination  of  NAC  data.  Two 
additional  items  where  other  DoD  organizations  report  higher  means  also 
relate  to  this  theme:  "degree  of  comprehensiveness  of  the  quality  plan 
within  the  organization,"  and  "extent  to  which  top  management  has 
objectives  for  quality  performance."  The  final  Item  In  this  category 
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Table  14 


Quality  Management  Items  Showing 
MAC  Self-Ratings  as  Lower^  than 
Other  DoD  Benchmark  Organizations 


MAC  DoD 

(N=ll)  (N=91) 

1.  Role  of  top  management  leadership  and  quality  policy 


2.8 

3.3 

comprehensiveness  of  the  goal -setting  process  for  quality 
within  the  organization 

2.8 

3.3 

extent  to  which  quality  goals  and  policy  are  understood 
within  the  organization 

3.0 

3.5 

degree  of  comprehensiveness  of  the  quality  plan  within  the 
organization 

3.0 

3.5 

degree  to  which  top  management  (commanding  officer, 
executive  director,  major  department  heads)  Is  evaluated 
for  quality  performance 

3.4 

3.9 

extent  to  which  top  management  has  objectives  for  quality 
performance 

3.  Training 

3.2  3.8  quality-related  training  given  to  non-supervisory 

employees  throughout  the  organization 


4.  Product/service  design 

2.9  3.6  quality  emphasis  by  customer  service  employees 


^  Differences  were  determined  using  a  t-test  comparison  of 
mean  ratings.  All  those  reported  are  significant  p<.10.  All  items 
not  reported  showed  no  difference  between  NAC  and  the  other 
participating  organizations. 
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Table  14  (cont'd) 


NAC  DoD 

{N=ll)  (N=91) 

5.  Supplier  quality  management 

1-8  2.3  Involvement  of  the  supplier  In  the  product  development 

process 


6.  Process  management 

2*7  3.2  clarity  of  work  or  process  Instructions  given  to  employees 

7.  Quality  data  and  reporting 

2-9  3.4  extent  to  which  quality  data  are  available  to  managers 

and  supervisors 


8. 

Employee  relations 

3.0 

3.6 

effectiveness  of  the  quality  circle  or  employee 
involvement  type  programs  in  the  organization 

3.0 

3.5 

effectiveness  of  supervisors  in  solving  problems/issues 

3.0 

3.5 

extent  to  which  employees  are  recognized  for  superior 
quality  performance 

3.5 

4.0 

extent  to  which  quality  circle  or  employee  Involvement 
type  programs  are  Implemented  in  the  organization 

2.7 

3.0 

degree  of  participation  In  quality  decisions  by  non- 
supervlsory  employees 

30 


shows  that  the  other  DoD  organizations  report  a  greater  use  of  quality 
performance  criteria  in  the  evaluation  of  top  management. 

The  next  five  categories  of  quality  management  each  include  only 
one  item  where  NAC's  self-rating  was  lower  than  the  comparison  group.  In 
all  cases,  the  items  were  identified  as  among  the  lowest  self-rated  in  the 
previous  discussion  of  NAC  results.  The  items  include:  the  extent  of 
quality-related  training  given  to  non-supervisory  employees,  the  quality 
emphasis  by  customer  service  employees,  the  involvement  of  the  supplier  in 
the  product  development  process,  and  the  clarity  of  work  or  process 
instructions  given  to  employees.  Again,  these  results  suggest  that  the 
comparison  data  is  redundant  to  the  discussion  of  NAC  data  alone. 

The  final  category  of  quality  management  is  employee  relations. 
In  this  category  there  were  four  items  that  had  lower  ratings  than  the 
comparison  group.  Only  one  of  these  items  was  identified  in  the  previous 
discussion  of  results  --  "degree  of  participation  in  quality  decisions  by 
non-supervisory  employees.  Two  of  the  remaining  items  deal  with  the 
extent  and  effectiveness  of  quality  circle  or  employee  involvement 
programs;  the  last  item  relates  to  the  effectiveness  of  supervisors  in 
solving  problems/issues. 

DISCUSSION:  HOH  CAN  THESE  DATA  BE  USED? 

As  described  in  the  introduction  to  the  presentation  of  results, 
there  are  no  norms  against  which  the  data  in  this  study  can  be  compared. 
The  primary  value  of  the  data  is  in  the  fact  that  they  represent  a  self- 
assessment  of  the  quality  management  practices  at  NAC  as  perceived  by  the 
top  management  group.  The  Applegate  study  (1991)  found  that  nonfinancial 
quality  measures  of  real-time  management  are  a  rare  commodity.  The  data 
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reported  here  can  serve  two  primary  purposes.  First,  they  can  be  used  to 
guide  discussion  among  the  executive  steering  group  regarding  actions  that 
might  be  planned  to  further  enhance  quality  at  NAC.  In  this  way,  the  data 
can  be  viewed  as  input  to  the  "check"  phase  of  the  PDCA  cycle.  The 
executive  steering  group  has  planned  and  implemented  numerous  continuous 
improvement  initiatives  within  the  organization  and  these  data  can  be  used 
as  a  form  of  assessment  of  progress  based  on  the  perceptions  of  the 
organizational  leaders.  The  variance  in  perceptions  may  illuminate 
activities  within  one  part  of  the  organization  that  are  not  well  known  by 
other  parts  of  the  organization;  or  may  identify  further  opportunities  for 
improvement  by  virtue  of  an  innovative  idea  of  one  of  the  steering  group 
members . 

The  second  way  in  which  these  data  may  be  used  is  as  a  baseline. 
As  the  organization  continues  its  quality  improvement  activities,  changes 
will  occur.  By  using  this  survey  with  the  same  sample  (or  an  expanded 
sample  including  Division  managers  or  a  other  organizational  members)  over 
time,  these  changes  can  be  tracked  and  the  PDCA  cycle  continuously 
evaluated.  This  study  is  not  intended  to  replace  the  need  for  more 
objective  indicators  of  quality  improvements;  however,  there  is  an 
important  potential  for  insight  into  opportunities  and  barriers  through 
the  discussion  of  these  data  by  the  organizational  decision  makers. 
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APPENDIX  A 


RESEARCH  PARTICIPANTS 


The  executive  steering  group  or  committee  at  each  of  the  following 
organizations  participated  in  the  thesis  survey.  A  point  of  contact  (POC) 
is  shown  for  each  organization  as  well  as  the  name  and  title  for  each 
organization's  interviewee. 


Sacramento  Air  Logistics  Center 
McClellan  Air  Force  Base 
Sacramento,  California 
Major  General  Michael  D.  Pavich,  USAF 
Center  Commander 
(POC  Colonel  Folz 
916-633-1164 
A/V  633-1164) 

Navy  Aviation  Supply  Office 
(formerly  Defense  Industrial 
Supply  Center) 

Philadelphia,  Pennsylvania 
Rear  Admiral  James  E.  Eckleberger,  USN 
Commanding  Officer 
(POC  Mr.  Marvin  Sandler 
215-697-1375 
A/V  442-1375) 

Naval  Avionics  Center 
Indianapolis,  Indiana 
Captain  Russell  J.  Henry,  USN 
Commanding  Officer 
(POC  Mr.  Thomas  Sibert 
317-353-7470 
A/V  369-7470) 

Naval  Aviation  Depot 
Naval  Station  Norfolk 
Norfolk,  Virginia 
Captain  Thomas  W.  Hancock,  USN 
Commanding  Officer 
(POC  Mr.  Ross  Haines 
804-445-1587) 

Norfolk  Naval  Shipyard 
Portsmouth,  Virginia 
Captain  James  T.  Taylor,  USN 
Commanding  Officer 
(POC  Mr.  Duff  Porter 
804-396-7092) 
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Naval  Ship  Systems 
Engineering  Station 
Philadelphia,  Pennsylvania 
Captain  Dennis  K.  Kruse,  USN 
Commanding  Officer 
(POC  Mr.  James  Summers 
215-897-7828) 

1926th  Communications-Computer  Group 
Warner  Robins  Air  Logistics  Center 
Warner  Robins  Air  Force  Base,  Georgia 
Mr.  Clifford  E.  Carroll 
Executive  Director 
(POC  Ms.  Jeanie  Spence 
912-926-7687 
A/V  468-7687) 

Naval  Supply  Center 
San  Diego,  California 
Captain  Gary  D.  Lynn,  USN 
Executive  Officer 
(POC  Ms.  Donna  Tierney 
619-532-1689 
A/V  522-1689) 

Naval  Aviation  Depot 
Marine  Corps  Air  Station 
Cherry  Point,  North  Carolina 
Mr.  John  C.  Adams 
TQM  Coordinator 
(POC  Mr.  John  Adams 
919-466-7403 
A/V  582-7403) 

Navy  Aviation  Supply  Office,  Code  10 
(formerly  Naval  Publications 
and  Forms  Center) 

Philadelphia,  Pennsylvania 
Lieutenant  Commander  Kenneth  K.  Kittredge,  USN 
Director,  Publications  and  Forms 
(POC  Mr.  Dennis  Cronin 
215-697-4919 
A/V  442-4919) 


34 


APPENDIX  B:  SURVEY  QUESTIONS 


Extent  or  Degree  of  Current  Practice  Is 
Very  Low  Low  Medium  High  Very  High 


Extent  to  which  the  top  12345 

executive  assumes  responsibility 
for  quality  performance 


Visibility  of  the  quality  12345 

department 


Specific  work-skills  training  12345 

(technical  and  vocational) 

given  to  non-supervisory  employees 

throughout  the  organization 

Thoroughness  of  new  process/  12345 

service  design  reviews  before 
the  process/service  is 
impl emented/produced 

Extent  to  which  suppliers  are  12345 

selected  based  on  quality 
rather  than  price  or  schedule 

Use  of  acceptance  sampling  to  12345 

accept/reject  lots  or  batches 
of  work 


Availability  of  cost  of  quality  12345 

data  in  the  organization 

Extent  to  which  quality  circle  12345 

or  employee  involvement  type 
programs  are  implemented  in 
the  organization 

Acceptance  of  responsibility  12345 

for  quality  by  major  branch/ 
department  heads  within  the 
organization 

Quality  department's  access  12345 

to  top  management 


1  2  3  4  5 


Quality-related  training  given 
to  non-supervisory  employees 
throughout  the  organization 
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xtent  or  Degree  of  Current  Practice  Is 
Very  Low  Low  Medium  High  Very  High 


Coordination  among  affected  12345 

departments  in  the  process/ 
service  development  process 

Thoroughness  of  the  12345 

supplier  rating  system 

Amount  of  preventive  12345 

equipment  maintenance 

Availability  of  quality  data  12345 

(error  rates,  defect  rates, 
scrap,  defects) 

Effectiveness  of  the  quality  12345 

circle  or  employee  involvement 
type  programs  in  the  organization 

Degree  to  which  top  management  12345 

(commanding  officer/executive 

director/major  department 

heads)  is  evaluated  for  quality 

performance 

Autonomy  of  the  quality  12345 

department 

Quality-related  training  12345 

given  to  managers  and 
supervisors  throughout  the 
organization 

Quality  of  new  processes/  12345 

services  emphasized  in  relation 
to  cost  or  schedule  objectives 

Reliance  on  reasonably  few  12345 

dependable  suppliers 

Extent  to  which  inspection,  12345 

review,  or  checking  of  work 
is  automated 

Timeliness  of  the  quality  12345 

data 

Extent  to  which  employees  12345 

are  held  responsible  for 
error-free  output 
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Extent  to  which  top  management 
supports  long-term  quality 
improvement  process 


Extent  or  Degree  of  Current  Practice  Is 
Very  Low  Low  Medium  High  Very  High 


Amount  of  coordination  12345 

between  the  quality  department 
and  other  departments 

Training  in  the  "total  12345 

quality  concept"(i .e.  philosophy 

of  organization-wide 

responsibility  for  quality) 

throughout  the  organization 

Clarity  of  process/service  12345 

specifications  and  procedures 

Amount  of  education  of  12345 

suppliers  by  the  organization 

Amount  of  incoming  inspection,  12345 

review,  or  checking 

Extent  to  which  quality  data  12345 

(cost  of  quality,  defects, 
errors,  scrap,  etc.)  are  used 
as  tools  to  manage  quality 

Amount  of  feedback  provided  12345 

to  employees  on  their  quality 

performance 

Degree  of  participation  by  12345 

major  branch/department  heads 
in  the  quality  improvement  process 

Effectiveness  of  the  quality  12  45 

department  in  improving  quality 

Training  in  the  basic  12345 

statistical  techniques  (such 
as  histograms  and  control 
charts)  in  the  organization 
as  a  whole 

Extent  to  which  implementation  12345 

/producibility  is  considered 
in  the  process/service  design 
process 
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Extent  or  Degree  of  Current  Practice  Is 
Very  Low  Low  Medium  High  Very  High 


Technical  assistance  provided 
to  suppliers 

Amount  of  in-process 
inspection,  review,  or  checking 

Extent  to  which  quality 
data  are  available  to  non- 
supervisory  employees 

Degree  of  participation  in 
quality  decisions  by  non- 
supervisory  employees 

Extent  to  which  top 
management  has  objectives 
for  quality  performance 

Training  in  advanced 
statistical  techniques  (such 
as  design  of  experiments  and 
regression  analysis)  in  the 
organization  as  a  whole 

Quality  emphasis  by  customer 
service  employees 

Involvement  of  the  supplier 
in  the  product  development 
process 

Amount  of  final  inspection, 
review,  or  checking 

Extent  to  which  quality  data 
are  available  to  managers  and 
supervisors 

Extent  to  which  quality 
awareness  building  among 
employees  is  ongoing 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 
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Extent  or  Degree  of  Current  Practice  Is 
Very  Low  Low  Medium  High  Very  High 


Specificity  of  quality  goals 
within  the  organization 

Commitment  of  the  top 
management  to  employee 
training 

Extent  to  which  longer  term 
relationships  are  offered  to 
suppliers 

Stability  of  production 
schedule/work  distribution 

Extent  to  which  quality  data 
are  used  to  evaluate  supervisor 
and  managerial  performance 

Extent  to  which  employees 
are  recognized  for  superior 
quality  performance 

Comprehensiveness  of  the 
goal-setting  process  for 
quality  within  the  organization 

Availability  of  resources  for 
employee  training  in  the 
organization 

Clarity  of  specifications 
provided  to  suppliers 

Degree  of  automation  of  the 
process 

Extent  to  which  quality  data, 
control  charts,  etc.,  are 
displayed  at  employee’s 
work  stations 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 
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Effectiveness  of  supervisors 
in  solving  problems/issues 


Extent  to  which  quality  goals 
and  policy  are  understood 
within  the  organization 

Extent  to  which  process  design 
is  "fool-proof  and  minimizes 
chances  of  employee  errors 

Importance  attached  to  quality 
by  top  management  in  relation 
to  cost  and  schedule  objectives 

Clarity  of  work  or  process 
instructions  given  to  employees 

Amount  of  review  of  quality 
issues  in  top  management  meetings 

Degree  to  which  top  management 
considers  quality  management  as 
a  way  to  increase  revenues/reduce 
costs 

Degree  of  comprehensiveness 
of  the  quality  plan  within 
the  organization 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 
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APPENDIX  C:  STATISTICAL  ANALYSIS  OF  SURVEY  RESULTS 

The  survey  used  in  this  study  is  adapted  from  an  instrument 
developed  and  validated  by  Saraph,  Benson  and  Schroeder  (1989).  The 
Saraph  et  al .  citation  provides  substantial  evidence  for  the  validity  of 
the  eight  critical  factors  of  quality  management  by  evaluating  content 
validity,  criterion-related  validity  and  construct  validity. 

For  this  research,  the  survey  was  modified  by  dropping  12  questions 
that  were  determined  to  be  unreliable  in  the  original  study.  As  stated  in 
the  body  of  this  report,  additional  modifications  to  wording  were  made  to 
fit  DoD  organizations.  The  modified  survey,  containing  66  questions,  was 
formally  reviewed  by  two  civilian  professors  of  management  in  order  to 
ensure  the  language  changes  would  ease  comprehension  of  the  survey 
questions  by  the  targeted  audience,  without  changing  the  substantive 
intent  of  the  questions. 

The  reliability  of  the  survey  data  collected  for  this  study  was 
evaluated  using  the  internal  consistency  method.  Cronbach's  alpha,  which 
is  well  suited  to  attitude  instruments  in  which  multiple  questions  are 
used  to  address  a  single  dimension  (i.e.  training,  process  management), 
was  chosen  to  assess  internal  consistency  reliability  (Jaeger,  1983).  The 
SPSS/PC+  reliability  program  was  used  to  conduct  the  analysis  (Norusis, 
1990).  Missing  data,  which  was  minimal,  was  handled  by  substituting  the 
median  score  for  each  survey  question,  so  as  not  to  exclude  any  survey 
responses  from  this  study. 

Results  for  the  eight  critical  factors'  reliability  are  detailed  in 
Table  3  (see  the  Methodology  section),  which  shows  that  the  reliability 
coefficients  or  alpha  scores  ranged  from  .73  to  .91,  all  of  which  are 
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considered  adequate  for  reliability  of  research  instruments.  This 
analysis  demonstrates  that  different  questions  intended  to  measure  the 
same  critical  factor  show  convergence  (Cronbach,  1951;  Jaeger,  1983;  Yin, 
1984).  These  results  further  supported  reliability  evidence  presented  by 
the  original  developers  of  the  instrument. 

A  correlation  matrix  for  the  critical  factors  of  quality  management 
was  completed  as  an  additional  measure  of  discriminant  validity,  and  is 
detailed  in  the  following  table.  Because  the  factors  all  deal  with 
quality  management,  significant  correlations  are  to  be  expected.  All  but 
four  intercorrelations  show  at  least  50%  unique  variance,  thus  supporting 
discriminant  validity.  The  highest  intercorrelation  was  found  between 
leadership  and  employee  relations  (r=.79).  This  suggests  that  these  two 
dimensions  have  62%  variance  in  common,  and  38%  unique  variance.  While 
this  is  not  a  strong  indication  of  discriminant  validity,  it  was  felt  to 
be  sufficient  for  purposes  of  this  study. 
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TABLE  C:  SCALE  TO  SCALE  CORRELATION  MATRIX  FOR  THE 

CRITICAL  FACTORS  OF  QUALITY  MANAGEMENT 

Critical  Factor  1 

2 

Scale  # 

3  4 

5 

6 

7 

8 

Role  of  management  1.0 

leadership  and  quality 
policy 
(scale  #1) 

.58 

.66  .71 

.31 

.46 

.72 

.79 

Role  of  the  quality 
department 
(scale  #2) 

1.0 

.49  .58 

.32 

.37 

.48 

.58 

Training 
(scale  #3) 

1.0  .56 

.42 

.43 

.66 

.66 

Product/service  design 
(scale  #4) 

1.0 

.40 

.58 

.71 

.69 

Supplier  quality 
management 
(scale  #5) 

1.0 

.59 

.42 

.37 

Process  management 
(scale  #6) 

1.0 

.64 

.46 

Quality  data  and  reporting 
(scale  #7) 

1.0 

.74 

Employee  relations 
(scale  #8) 

1.0 
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